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Abstract 

In the process of evaluating online learning products for accessibility, researchers in the Center on Online Learning and Students 
with Disabilities concluded that most often consultation guides and assessment tools were useful in determining sensory 
accessibility but did not extend to critical aspects of learning within the Universal Design for Learning (UDL) framework. To 
help fill this void in assessment, researchers created the UDL Scan tool to examine online learning products alignment to the UDL 
framework. This article provides an overview of how accessibility has been historically measured and introduces the need to 
move beyond the traditional understanding of accessibility to a broader UDL-based lens. With this understanding, a UDL Scan 
tool was developed and validated to investigate the alignment of online learning content to UDL. This article will present the 
process of development, the validation, and discuss how the measurements provide critical benchmarks for educators and 
industry as they adopt new online learning systems. 


Keywords 

Universal Design for Learning, instructional/policy perspectives, K-12 online learning, accessibility 


Introduction 

Although blended and fully online K-12 learning opportunities 
have grown in popularity, investigations into the central com¬ 
ponent of online learning are limited. Although there are a 
number of online and blended learning models (Christensen, 
Horn, & Stalcer, 2013) that alter the online learning experience 
for students, the constant design feature is that the significant 
majority (up to 90%) of K-12 online learning is instructed via 
prepackaged content and/or curriculum (Patrick, Kennedy, & 
Powell, 2013). Thus, unless districts or teachers invest time in 
designing learning experiences tailored to the individual lear¬ 
ners, students are learning from materials that are likely devel¬ 
oped by an outside vendor. Learners in blended or fully online 
environments interact with these prepackaged materials 
throughout their entire instructional experience, often from ini¬ 
tial instruction through assessment. 

As highlighted in Smith and Basham (2014), the role of the 
teacher in K-12 online environments is different from that of a 
traditional brick-and-mortar teacher. With the wide variety of 
K-12 online environments (e.g., fully online, blended, supple¬ 
mental, personalized), the role of the teacher varies based on a 
number of factors associated with the learning environment. 
Minimally the learning environment is comprised of the lear¬ 
ner, the adopted online system or various systems, the physical 
environment (e.g., an active classroom, a computer lab with 
100 other students, a desk at home, a kitchen table, a couch 


at home), other individuals, if any, within the environment 
(e.g., adults, other learners, caregivers). Importantly, depend¬ 
ing on the online learning model, adopted online system, and 
the expectations of the environment, research has indicated that 
often the primary role of a traditional teacher as the instructor is 
replaced by that of an online system (Rice & Carter, 2015a, 
2015b). 

What is unknown by those outside K-12 online education is 
that school districts and classroom teachers typically do not 
develop their own lessons for many online environments (Rice 
& Carter, 2015a; Smith & Basham, 2014). The investment of 
time and resources required by school districts or the classroom 
teacher to create online content is often simply too prohibitive. 
The development of online curriculum and discipline-specific 
content places additional demands on resources that are often 
already overwhelmed. Instead, the materials are typically 
developed by and purchased from vendors who offer prepack¬ 
aged learning products at a more reasonable cost. 

These online products come in the form of digital lessons, 
activities, and resources, structuring the learning experience 
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and directing what the student completes on a daily basis and 
across the entire course. The teacher is the instructor of record, 
but the vendor-based digital lesson and digital system drive the 
learning experience through specific lessons, activities, accom¬ 
panying assessments, and the predetermined path for subse¬ 
quent lesson completion (Basham, Stahl, Ortiz, Rice, & 
Smith, 2015; Rice & Carter, 2015a). In essence, the digital 
lesson/material offers the actual learning experience for many 
blended and fully online learners, and any teacher actions sup¬ 
plement this experience (Rice & Carter, 2015a, 2015b). 

Although the role of the online teacher may be disturbing for 
some, it is not the primary focus of this article. Of course, both 
the roles of the teacher and the online system are dynamic, 
based on environmental factors as well as innovations in tech¬ 
nology (e.g., machine learning, artificial intelligence, intelli¬ 
gent agents). Nonetheless, the transformation of the teacher’s 
role in the online environment provides credence for further 
research in a number of areas. For instance, although K-12 
online learning has received increased attention with research 
examining student outcomes, the examination of the prepack¬ 
aged online content—the primary element of the K-12 online 
learning experience—has not received adequate attention 
(Smith & Basham, 2014). Research on the effects of prepack¬ 
aged digital materials on student learning, specifically for the 
struggling learner and those with identified disabilities, is not 
represented in current research efforts. 

Through research conducted in the Center on Online Learn¬ 
ing and Students with Disabilities (Center), this article high¬ 
lights the review of K-12 digital learning curricula and content 
within online learning systems. The article begins with a brief 
overview of accessibility guidelines for digital materials. We 
then describe some of the limitations for using only these stan¬ 
dards in determining effectiveness of online learning curricula 
and associated content for students, especially for students with 
disabilities. Specifically, it is argued that using the Universal 
Design for Learning (UDL) framework as specified in the 
Every Student Succeeds Act (ESSA, 2015) along with current 
accessibility guidelines provides a stronger basis for the review 
of online learning materials. Finally, the description, develop¬ 
ment, and validation of a measurement tool used to measure the 
alignment of online learning systems to the UDL framework is 
presented. It is hoped this article will encourage further 
research and dialogue about the design and implementation 
of digitally driven K-12 learning environments for students 
with disabilities. 

Understanding Accessibility 

As growth in K-12 online learning experiences has increased, 
so has the number of struggling students and their peers with 
disabilities who are enrolled in online learning (Basham, 
Smith, Greer, & Marino, 2013 ). The inclusion of these students 
in both blended and fully online courses has demanded reflec¬ 
tion and reconsideration of the appropriateness of the content 
and overall instruction. The recent policy scan presented in the 
Center’s publication, Equity Matters: Digital and Online 


Learning for Students with Disabilities, noted that only 36% 
of states guarantee that their K-12 environments are accessible 
for students with disabilities (Basham et al., 2015). Moreover, 
the lack of required data, as well as data sharing, on students in 
these online environments makes it difficult to ascertain the 
impact of these prepackaged learning materials on student out¬ 
comes. Thus, the growth in numbers, combined with the lack of 
guaranteed accessibility, requires a determination in the acces¬ 
sibility and, more importantly, usability, and even leamability 
of these digital materials, lessons, activities, and assessments 
for all students. 

Accessibility Standards and Guidelines 

The rights of all users to access digital content actually precede 
the recent trend in K-12 blended and fully online learning. In 
the United States, the amended Section 508 (1998) of the 
Workforce Rehabilitation Act of 1973 enhances access to 
broadband (e.g., Internet, online learning) technology and ser¬ 
vices for individuals with disabilities. Additional standards 
have followed, including the World Wide Web Consortium 
(W3C) accessibility guidelines for the W3C’s Web Accessibil¬ 
ity Initiative (2014) and the International Digital Publishing 
Forum’s (1DPF) EPUB (2014) content publication standards. 
Outside the United States, the European Unified Approach for 
Accessible Lifelong Learning (EU4ALL, 2010) initiated the 
concept of accessible lifelong learning and the elimination of 
barriers to the interlinked worlds of education and work 
through the use of appropriate digital technologies. 

There are two primary definitions that are frequently used in 
defining web or digital accessibility. They include (1) accessi¬ 
bility means that people with disabilities can use the web— 
people with disabilities can perceive, understand, navigate, 
interact, and contribute with or to the web (World Wide Web 
Consortium, 2005) and (2) technology is accessible if it can be 
used as effectively by people with disabilities as by those with¬ 
out (Yesilada, Brajnik, Vigo, & Harper, 2012). These defini¬ 
tions, combined with the standards and guidelines, shape the 
current measures used by digital material developers and 
school district personnel to determine whether K-12 content 
is appropriate for those with disabilities. 

The application of the accessibility standards has sought to 
promote accessible digital designs for materials and navigation 
of the learning system. For example, in an applied sense, the 
standards promote design tips which include providing the text 
equivalent or closed captioning for animation and video con¬ 
tent, color contrast and appropriate font size, transcripts of all 
audio and accompanying descriptions for any video, and fre¬ 
quent accessibility testing during and after the digital content 
development and overall course design (W3C, 2014). These 
features target alternate means of accessing the digital materi¬ 
als. The standards then require an alternate format, for exam¬ 
ple, supporting an audio file with a complete transcript or 
closed captioning the audio portion of a video. 

Focused on ensuring the online K-12 marketplace had a 
minimal standard of accessibility, pioneers such as Rose 
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(2007) wrote a report for the International Association for 
K-12 Online Learning (iNACOL) calling on developers and 
providers to meet the basic accessibility standards. With a pri¬ 
mary focus on sensory and physical accessibility, Rose focused 
his report on the Office of Civil Rights’ (OCR) definition of 
accessibility which extends Section 508 guidelines to technol¬ 
ogy accommodations in order to access educational opportu¬ 
nity, and do so in a timely manner. OCR clarified the specific 
legal requirements specific to digital curriculum, which applies 
to the K-12 blended and online classroom by stating: 

equal opportunity, equal treatment, and the obligation to make 
accommodations or modifications to avoid disability based dis¬ 
crimination—also apply to elementary and secondary schools 
under the general nondiscrimination provisions in Section 504 and 
the ADA. The application of these principles to elementary and 
secondary schools is also supported by the requirement to provide a 
free appropriate public education (FAPE) to students with disabil¬ 
ities. (OCR, 2011). 

Although the OCR guidance document ensures that digital 
materials, delivery systems, and devices are accessible, the 
parameters of accessibility are restricted to sensory and phys¬ 
ical consideration. 

In an updated report for iNACOL, Rose (2014) again primar¬ 
ily focused on sensory and physical accessibility. While the 
report makes reference to UDL, the emphasis is on the accessi¬ 
bility portions of the UDL guidelines with the foundational focus 
on Section 504 and 508 provisions for digital information, with 
an added reference for access determinations to be based on 
W3C’s Web Content Accessibility guidelines. Recommenda¬ 
tions, for example, suggest that OCR alignment constitutes 
closed captioning for animation and video products, tagging all 
graphics with corresponding text, carefully selecting and using 
color, and ensuring that all graphics have defined alt tags to 
allow for screen reader access. These approaches reinforce an 
accessibility evaluation process targeting a limited population of 
individuals who require these features or modifications. 

To provide developers and educators with guidance in deter¬ 
mining digital accessibility (especially alignment to Section 
508 expectations), Hashey and Stahl (2014) introduced the 
Voluntary Product Accessibility Template (VPAT). Created 
to share specific product accessibility information with educa¬ 
tors and other professionals seeking to acquire accessible digi¬ 
tal materials, the VPAT examines devices, software, and digital 
materials to better determine how these materials can be used 
by those with visual impairments, hearing impairments, or lim¬ 
ited mobility. The VPAT provides a thorough and detailed 
overview of the digital product and can make comparisons 
much easier for the user to understand and apply when making 
accessibility decisions. 

Measuring Accessibility 

As noted in Hashey and Stahl (2014) as a Center resource, the 
VPAT table was created to offer a quick review of more than 70 


products used in K-12 online learning (see http://centerononli 
nelearning.org/resources/vpat/). The Center’s review, Quick 
Guide to Accessible Products in Education , offers a visual 
reference to the extent to which each product is accessible. The 
interactive VPAT table available through the Center’s website 
offers educators, developers, and other interested parties an 
understanding (or at least a starting point) of how to determine 
whether a product is appropriate for the K-12 learner, espe¬ 
cially those with sensory and physical disabilities. As a stan¬ 
dard, the VPAT does not provide for the majority of students 
with disabilities who have cognitive, learning, attention, or 
behavioral needs. 

Moving Beyond Traditional Accessibility 

Traditional accessibility concentrates on multiple formats but 
not alterations to the learning demands of the digital material. 
For example, providing an accessible digital text (e.g., online 
textbook, digital text-based lesson) often requires formatting 
the digital text to allow for a text-to-speech application to 
automatically read the text for individuals with print impair¬ 
ments (e.g., someone who is blind and cannot see the text). 
Accessibility, in this instance, does not measure for the poten¬ 
tial of supports to encourage greater readability of text (Flesch, 
1948; Mosenthal & Kircsh, 1998; Valencia, Wixson, & 
Pearson, 2014) or the ability to match digital text content to 
individual learners, based on actual readability and other asso¬ 
ciated metadata (Denning, Pera, & Ng, 2016). Moreover, this 
traditional understanding of accessibility, beyond sensory 
accessibility, neglects to identify other critical elements for sup¬ 
porting overall learning and comprehension. For instance, these 
scans do not measure potential for engagement (O’Brien & 
Toms, 2008), ability to resize the amount of text in lines 
(Schneps, Thomson, Chen, Sonnert, & Pomplun, 2013), supports 
for reducing the demands of content-specific vocabulary (Nagy 
& Townsend, 2012), use of multiple forms of media in digital 
content such as interactive simulations (Schneps et al, 2014). 

Thus, traditional accessibility standards for digital materials 
address sensory and physical challenges, but are limited in how 
they support cognitive and learning barriers experienced by 
individuals with identified disabilities, along with their peers 
who may not have an identified disability but who struggle with 
reading, processing, memory, and similar cognitive demands 
associated with learning. Moreover, traditional notions of 
accessibility assume an intermediary will interact with content 
to make it more usable and/or that a teacher will use the mate¬ 
rials in such a way to support the learning process. Unfortu¬ 
nately, as aforementioned, within many K-12 online learning 
environments teachers have very little control over the actual 
content, its delivery, and associated instruction within the sys¬ 
tem. Thus, wherein accessibility standards are an important 
starting point for considering online content, the current under¬ 
standings and assumptions of accessibility fall short when con¬ 
sidering the reality of K-12 online learning practice. 

In order to ensure accessibility for learners, the cognitive 
accessibility and leamability of content within an associated 
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tool should also be evaluated. Because most school districts are 
purchasing prepackaged curriculum and content from vendors, 
stakeholders in the purchase of those products need the relevant 
guidelines and tools to evaluate those products for appropriate¬ 
ness and accessibility for learners. While the Center’s VP AT 
evaluation of vendor-developed K-12 online learning products 
were effective in ensuring that the products conformed to basic 
accessibility guidelines and policies, this conformance alone 
falls short of assessing content and the associated systems for 
supporting usability or leamability of content in the learning 
process. As a starting point for assessing this usability, Center 
researchers determined that an analysis of K-12 blended and 
fully online learning should employ an evaluation of content 
adherence to the UDL framework. To measure accessibility 
using the UDL framework, Center researchers developed the 
UDL Scan tool to measure alignment of online learning content 
and associated systems to the principles, guidelines, and 
checkpoints. 

Enter a Broader Understanding of UDL 

UDL is an instructional framework that is based on substantial 
amounts of scientifically based research (e.g., Dalton, Proctor, 
Uccelli, Mo, & Snow, 2011; Kennedy, Thomas, Meyer, Alves, 
& Lloyd, 2014; Marino, 2009; Proctor et al., 2011; Rappolt- 
Schlichtmann et al. 2013). As a scientifically based framework, 
UDL works to support the variability of all learners by both 
proactively and iteratively designing learning with a focus on 
the integration of providing multiple means of engagement, 
representation of information, and action and expression of 
understanding. The framework is defined within the Higher 
Education Opportunity Act (HEOA, 2008): 

... [UDL is] a scientifically valid framework for guiding educa¬ 
tional practice that—(A) provides flexibility in the ways informa¬ 
tion—is presented, in the ways students respond or demonstrate 
knowledge and skills, and in the ways students are engaged; and 
(B) reduces barriers in instruction, provides appropriate accommo¬ 
dations, supports, and challenges, and maintains high achievement 
expectations for all students, including students with disabilities 
and students who are limited English proficient. 

More recently, UDL was highlighted in the ESSA (2015) as 
well as the National Educational Technology Plan (NETP, 
2016) as a basis for designing as well as implementing learning 
environments, systems, and assessments for all learners, espe¬ 
cially learners with disabilities. Specifically, the language in 
ESSA indicates that districts should ensure that use of technol¬ 
ogy is not only accessible for all learners, but that systems also 
align to the UDL framework. 

As highlighted in Rose (2014), UDL is often viewed only in 
terms of accessibility. In reality, the framework of UDL pro¬ 
vides a much broader perspective than accessibility alone. As 
highlighted by CAST (2011) in the UDL guidelines, the frame¬ 
work moves from ensuring basic accessibility to an advanced 
or even metacognitive state of learning. This is evident by 


viewing the guidelines from either top to bottom or bottom to 
top (depending on the version of the guidelines). In the tradi¬ 
tional print edition of the guidelines (with the principle of 
representation to the left) under Providing Multiple Means for 
Representation, the guidelines move from perception (sen¬ 
sory), to clarifying and decoding information (basic learning 
input), and finally to supports for comprehension and general¬ 
ization of information (more advanced learning). Basham and 
Marino (2013) and Rappolt-Schlichtmann et al. (2013) discuss 
how UDL can be applied in broader perspectives than simply 
accessibility. 

As highlighted in Basham and Marino (2013), UDL applies 
an engineering-based perspective to the way learning environ¬ 
ments, curriculum, instruction, instructional tools, and assess¬ 
ment are both designed and utilized. Specifically, UDL-based 
instruction should consider the four critical elements of UDL 
instruction (2011); these elements include having clear goals, 
inclusive and intentional planning for variability, providing 
flexible methods and materials, and timely progress monitor¬ 
ing. These elements can be integrated into a five-step backward 
instructional design process: 

Step 1: Establish clear outcomes 

Step 2: Anticipate the learner variability 

Step 3: Design measurable outcomes and assessment plan 

Step 4: Design the instructional experience 

Step 5: Evaluate and reflect on new understandings 

To assist districts and teachers in the implementation of 
UDL, there must be guidance provided in how associated 
instructional materials and systems support UDL-based 
instruction. If teachers understand the type of learner variabil¬ 
ity that specific products could account for, they could then 
consider this understanding in the instructional design and 
implementation process. In a basic example, if a teacher knew 
a product only supported content understanding in English and 
she had learners that primarily learned in Spanish, then she 
would know there was a need to find a different product or 
take other measures to support representation of content. Thus, 
any guidance on UDL alignment would require the measure¬ 
ment of UDL in product systems. 

Unfortunately, there have been minimal attempts in measur¬ 
ing UDL as an entire framework. In fact, Basham and Gardner 
(2010) discussed the complexity of measuring UDL as a design 
framework, rather than a specific strategy or practice that can be 
easily observed in the environment. They indicated that any tool 
attempting to measure UDL would have to be multifaceted and 
measure the proactive design as well as the implementation 
within an instructional environment. Given that online learning 
tools are often being proactively designed and developed sepa¬ 
rate from instruction, measuring the design of these products for 
alignment to UDL is a necessary step in ensuring that districts 
and teachers know that they are adopting tools that are not only 
accessible but that also support the implementation of UDL. 

Researchers at the Center undertook developing a tool that 
could investigate online learning product alignment to UDL, 
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one that focused on developing such a measurement tool. Spe- Table I. UDL Scan Tool Items in Correspondence to UDL 
cifically, this project sought to answer: Checkpoints. 


(1) Can a UDL Scan tool be developed and validated, one 
that adequately measures the alignment of an online 
instructional product or system to the UDL 
framework? 

(2) Using a UDL Scan tool, what is the usability or feasi¬ 
bility of conducting product scans? 

Methods 

The development and validation of a tool to measure content 
and curriculum alignment to the UDL framework required a 
multiphase design involving (a) item generation, (b) pilot 
review, (c) content validation, and (d) an assessment of relia¬ 
bility and construct validity. 


Development of the Tool 

Initial development of the UDL Scan tool began with an anal¬ 
ysis of the UDL principles, guidelines, and checkpoints as well 
as a review of existing rubrics and observation instruments. To 
create a tool appropriate for evaluating online learning prod¬ 
ucts, the developers of the UDL Scan tool who are known 
experts in UDL met with other known experts, including senior 
personnel at CAST, to discuss the components that should 
comprise the tool. Based on these initial meetings, the devel¬ 
opers crafted evaluation questions, organized around the UDL 
guidelines and checkpoints, to identify whether UDL-based 
features were present within a product (and to what degree). 
As the tool was revised and refined, the developers continued to 
seek feedback from the UDL experts to ensure that the tool 
comprehensively assessed the three primary principles, nine 
guidelines, and numerous checkpoints of the UDL framework. 
This process involved a thorough consideration of the purpose 
of each of the principles, guidelines, and checkpoints, consid¬ 
ering the stated text, the intent of the text, examples of the text, 
and how it would be applied in the field, especially in the area 
of blended and fully online learning. From these examinations, 
items were developed to ensure correspondence to elements in 
the UDL framework (see Table 1). 

The UDL Scan tool was created using Qualtrics Labs, Inc. 
software, Version 12.018 of the Qualtrics Research Suite 
(Qualtrics Labs, 2012). Using the Qualtrics software allowed 
the developers to make the evaluation tool online accessible for 
users. It also allowed the developers to employ skip logic in the 
survey, to ensure greater usability and ease in use. Essentially, 
depending on how primary questions were answered, follow-up 
questions would only be asked as applicable. Because the UDL 
Scan tool was developed online, evaluators could explore and 
test the product while also answering questions about the prod¬ 
uct in a separate browser window. 

Once the initial questions were developed, thorough test¬ 
ing was conducted. To test the evaluation tool, the devel¬ 
opers met as a group to practice using the UDL Scan tool to 


UDL Scan Tool 

Items UDL Checkpoints 


Items 1-9 
Items 10-26 
Items 27-42 
Items 43-45 
Items 46-54 
Items 55-57 
Items 58-60 
Items 61-64 

Items 65-69 
Items 70-73 
Items 74-76 
Items 77-81 

Items 82-84 

Items 85-87 
Items 88-94 
Items 95-97 

Items 98-100 

Items 101-105 
Items 106-108 
Items 109-1 I I 
Items I 12-1 14 
Items I 15-1 17 
Items I 18-120 
Items 121-125 
Items 126-128 

Items 129-131 
Items 132-134 
Items 135-137 

Items 138-140 
Items 141-143 
Items 144 
Items 145-146 


Demographics 

1.2 Offer alternatives for auditory information 

1.3 Offer alternatives for visual information 

4.1 Vary the methods for response and navigation 

2.1 Clarify vocabulary and symbols 

2.2 Clarify syntax and structure 

2.3 Support decoding of text, mathematical 
notation, and symbols 

2.4 Promote understanding across languages 

2.5 Illustrate through multiple media 

3.1 Activate or supply background knowledge 

3.2 Highlight patterns, critical features, big ideas, 
and relationships 

3.3 Guide information processing, visualization, and 
manipulation 

3.4 Maximize transfer and generalization 

5.1 Use multiple media for communication 

5.2 Use multiple tools for construction and 
composition 

5.3 Build fluencies with graduated levels of support 
for practice and performance 

6.1 Guide appropriate goal setting 

6.2 Support planning and strategy development 

6.3 Facilitate managing information and resources 

6.4 Enhance capacity for monitoring progress 

7.1 Optimize individual choice and autonomy 

7.2 Optimize relevance, value, and authenticity 

7.3 Minimize threats and distractions 

8.2 Vary demands and resources to optimize 
challenge 

8.3 Foster collaboration and community 

8.4 Increase mastery-oriented feedback 

9.1 Promote expectations and beliefs that optimize 
motivation 

9.2 Facilitate personal coping skills and strategies 

9.3 Develop self-assessment and reflection 
Ease of use 

4.2 Optimize access to tools and assistive 
technologies 


Note. UDL = Universal Design for Learning. 


evaluate two online learning products. The walkthrough 
allowed the developers to identify features that were not 
adequately assessed and to troubleshoot the tool during use. 
Revisions were made to ensure that the questions asked 
through the tool both clearly and adequately assessed 
whether UDL features were available in the products being 
evaluated during testing. 

This extensive testing allowed the developers to more nar¬ 
rowly define the scope of the UDL Scan tool. Although the tool 
has potential value for evaluating isolated learning manage¬ 
ment systems (LMS) used to house content (e.g., Blackboard), 
the developers instead chose to focus the initial tool on prod¬ 
ucts that provide instructional content. 
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To accompany the UDL Scan tool, developers created a 
training manual for users. The manual provides detailed infor¬ 
mation about the expectations of the reviewer and how to use 
the tool, descriptions and examples of UDL features, and a 
glossary of the terms that appear in the UDL tool. The manual 
serves as a guide for teaching evaluators how to use the tool and 
as a resource for evaluators. 


The Instrument 

The UDL Scan tool provides researchers and educators with a 
measurement tool to review online content systems for their 
potential to support learner accessibility and variability. Each 
UDL guideline and checkpoint was mapped to specific features 
within a content system. Each item on the UDL Scan tool aligns 
with one of UDL’s three principles, nine guidelines, and at least 
one checkpoint, measuring each of them for each lesson 
evaluated. 

The scan tool consists of 37 initial items with a total of 46 
unique response items, including a tool for measuring product 
usability. The tool intuitively branches users to the specific 
questions they need for a thorough evaluation of the materials 
being scanned. If the tool is completed in its entirety (all 
branching items), there are a total of 146 items. 

The UDL Scan tool consists of multiple choice and Likert- 
type scale questions. Answers are submitted online. The scan 
tool begins with a series of questions designed to gather infor¬ 
mation about the evaluator, including the type of browser the 
evaluator is using to examine the product. The initial questions 
also gather information about the types of products and lessons 
being evaluated. The subsequent questions are broken into sec¬ 
tions associated with the UDL principles, guidelines, and 
checkpoints. Each section begins with a multiple-choice ques¬ 
tion designed to determine whether a product has features that 
incorporate a specific UDL checkpoint. If the evaluator deter¬ 
mines the product does include, or might include, features 
related to that checkpoint, the evaluator is provided with more 
specific questions to identify the degree to which those features 
are accessible. However, if the evaluator determines that the 
UDL checkpoint is not a part of the product, the scan tool is 
designed to move the evaluator to the next UDL checkpoint so 
as to avoid asking the evaluator irrelevant questions. The ques¬ 
tions are designed to determine how frequently aspects of a 
feature are available and to pinpoint specifically which exam¬ 
ples of a feature are accessible to the users. For example, one 
question asks the evaluator to indicate on a Likert-type scale 
how frequently the product illustrates content through videos, 
audio, and still images. 

Along with assessing the extent of UDL features available 
within online learning products, the scan tool also is used to 
assess the usability of the product. Specifically, the UDL Scan 
tool includes a set of Likert-type scale questions adapted from 
the System Usability Scale (Brooke, 1996). These questions are 
designed to measure how easy, or complex, a specific product 
is to use. 


Procedures 

To assess the interrater reliability of the scan tool, the following 
procedures were followed. Three graduate research assistants 
evaluated three product systems. Ten lessons within each of 
those product systems were randomly chosen and then evalu¬ 
ated for a total of 30 lessons across three products. Prior to 
evaluating the product systems, the research assistants attended 
a training session to learn how to use the Scan tool. During this 
training session, the trainer demonstrated how to access and use 
the tool and the online learning product systems being evalu¬ 
ated. The trainer also reviewed the training manual with the 
evaluators to ensure that the reviewers understood what was 
expected of them. During the training, the graduate research 
assistants were given an opportunity to ask questions, explore 
the online learning products, and practice using the scan tool. 
They also received a copy of the training manual to serve as a 
resource while using the Scan tool. 

Data Analysis 

Having three different raters utilize the scan tool across three 
product systems and 30 different lessons allowed for assess¬ 
ment of the interrater reliability. The interrater reliability anal¬ 
ysis measured whether the three graduate research assistants 
evaluated each of the lessons in the same way. Krippendorff s a 
(Krippendorff, 2004) and Fleiss’s k (Fleiss, 1971) were calcu¬ 
lated to determine reliability among raters. When 100% agree¬ 
ment was achieved across all three raters for each of the 10 
lessons in the system, Krippendorff s a and Fleiss’s k could not 
be calculated due to no variation in the ratings. Because the 
cases of perfect agreement were not included in calculating the 
mean and median reliability values, they were biased toward 
less agreement. The degree of bias is not known and cannot be 
measured. Fleiss’s k was calculated only when there was no 
missing rating; this amounted to most the cases in the initial 
system and some cases in the subsequent systems. 

Results 

Interrater Reliability 

In general, interrater reliability was supported in each of the 
three systems—a high percentage of agreement (66-71% in 
total, median = 90-100%) was observed; and Fleiss’s k was 
greater than .20 (mean = .26-40, median = .28-40), suggest¬ 
ing fair agreement among the three raters (see Table 2). When 
only the initial items were examined, the total percentage of 
agreement slightly increased but the k values dropped, but still 
fair in the System 1 and System 2. However, Krippendorff s a 
was very low in all the three systems (e.g., negative mean and 
median a values), suggesting that disagreement between the 
three raters is systematic and, therefore, greater than what can 
be generally expected by chance (Krippendorff, 2004). In real¬ 
ity, disagreement was not systematic due to the small number 
of digital lessons evaluated and overlap of design principles 
characteristic in these materials. 
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Table 2. Interrater Reliability Within and Across All Three Product Systems. 

Percentage of Agreement Percentage of Agreement Krippendorff a Krippendorff a Fleiss k Fleiss k 
System Scanned (Total) (Median) (Mean) (Median) (Mean) (Median) 


System I 


All items 

71 

93 

-.09 

.00 

.26 

.28 

Initial items only a 

77 

92 

-.03 

.00 

.25 

.27 

System 2 

All items 

73 

100 

-.26 

-.32 

.32 

.40 

Initial items only 3 

81 

97 

-.10 

-.12 

.26 

.28 

System 3 

All items 

66 

90 

-.26 

-.29 

.40 

.36 

Initial items only 3 

71 

83 

-.19 

-.14 

.24 

.18 

Across all three systems 

All items 

70 

81 

-.02 

-.04 

.22 

.17 

Initial items only 3 

76 

78 

.13 

.00 

.24 

.12 


“Thirty-seven initial items that were always rated regardless of system. 


Discussion 

Across the country, online learning is growing at a rapid pace 
and a majority of the instructional products and tools used in 
these environments are purchased prepackaged from a vendor 
(Smith & Basham, 2014). The current focus on the traditional 
understanding of accessibility is critical to ensuring that learners 
with sensory and physical disabilities have basic access to these 
digital learning materials. Regrettably, while basic accessibility 
is still a need, current understanding of accessibility does little to 
support actual learning, especially in consideration of things 
such as cognitive accessibility. Within the United States, recent 
legislation (specifically ESS A, 2015) supports the need for dis¬ 
tricts to consider the implementation of UDL in the way they 
design and implement instruction. Understanding this need, this 
project sought to develop and test a tool for measuring a digital 
instructional product’s alignment to the UDL framework. 

The outcomes of the project indicate that the UDL Scan tool 
demonstrated success in the measurement of UDL within digital 
instructional products. Moreover, the tool was developed in part¬ 
nership with CAST, the founders of UDL. Thus, as an initial 
measurement tool, the UDL Scan tool demonstrates potential for 
measuring UDL alignment in digital instructional products. 
Potential uses of this tool include release for wider consumption 
and use. Specifically, by providing districts access to this tool, 
they will be able to evaluate products during the product acqui¬ 
sition process. As a result, it is hoped that districts will make 
more informed decisions in the procurement of digital instruc¬ 
tional products. It is foreseen that product developers may also 
use such a tool to support the design and eventual self-report of a 
product’s alignment with UDL. Optimistically, supporting UDL 
alignment will advance the field’s understanding and acceptance 
of basic accessibility to a more advanced consideration of build¬ 
ing product and systems with a focus on all learners. 

Implications for Practice 

Through the use of the UDL Scan tool, teachers have the poten¬ 
tial to develop a more nuanced understanding of learner 


variability and how tools associated with instructional practices 
may help adequately support this variability. Moving beyond 
the basic understandings of accessibility, teachers also have the 
ability to take a larger role in ensuring all learners are actively 
engaging and demonstrating the desired outcomes in the learn¬ 
ing process. Specifically, teachers can make better informed 
decisions about how to design, implement, and test learning 
experiences that meet the needs of the individual learners. 
From a teacher development perspective, this would advance 
a teacher’s need to more fully understand the conceptual, prac¬ 
tical, and testable underpinnings of UDL and the instructional 
design process, thus enhancing a teacher’s ability to take on the 
mind-set and operational status of a learning engineer (Basham 
& Marino, 2013). 

Implications for Future Research 

The development and validation of the UDL Scan tool advances 
the field’s ability to more adequately assess and research the 
framework of UDL. Since conducting this initial study, the UDL 
Scan tool has been used within the Center to measure alignment 
on more than 1,000 individual pieces of content. Using the tool, 
researchers have measured alignment of certain popular, 
blended, and fully online content to the UDL framework (Smith, 
2016). The goal was to help understand whether vendor-created 
K-12 online lessons were both accessible and appropriate for all 
students, especially those with disabilities. Finally, a next step 
would be to measure the instructional experience within these 
products. While a product may have alignment (or lack thereof) 
to UDL, there is also need to measure how a product provides 
actual instruction. Such an addition to the UDL Scan tool would 
allow users to evaluate whether online instructional systems 
(e.g., K12, Khan Academy) align to evidence- and/or research- 
based instructional practices. 

Limitations 

This study sought to research and test a UDL Scan tool for 
measuring the basic alignment of a digital instructional product 
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to UDL. The Scan tool was tested with digital products that 
provided students with instructional materials rather than an 
LMS (e.g., Blackboard). Thus, the UDL Scan tool was not 
designed or tested in the ability to measure an LMS or Content 
Management System (e.g., Wordpress) without embedded con¬ 
tent or a designed instructional sequence. The tool also was not 
designed to measure a brick-and-mortar instructional lesson. 
Users would be cautioned in attempting to measure alignment 
of any instructional experience beyond the intended use of the 
tool. Finally, although the UDL Scan tool has demonstrated 
consistent findings across further scans, this initial study only 
used 30 instructional lessons, thus some caution must be con¬ 
sidered given small number of these lessons when interpreting 
the mean and median of a and k values. 

Conclusion 

The ability to move the field of K-12 online learning beyond 
the basic understandings of accessibility to a more advanced 
understanding of UDL will support better online learning mate¬ 
rials for all learners, especially those learners with disabilities. 
As the K-12 education system moves increasingly online, it 
becomes more dependent on the educational technology indus¬ 
try to support the design of digital instructional materials and 
experiences. Thus, it is important for educators, researchers, 
and the industry to develop a shared understanding as well as 
expectations for these online materials and systems. The UDL 
framework provides a foundational structure for developing 
this shared understanding and using a tool, such as the UDL 
Scan tool, provides initial support for this cooperative effort. 
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